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ABSTRACT 

This study investigates the validity of second 
language vocabulary tests r and whether or not test scores accurately 
reflect the quality of language behavior in real-life situations. It 
attempts to prove the validity of vocabulary testing by comparing 
test scores to indices of vocabulary richness in second language 
production. The student subjects were assigned an essay on a 
designated topic. Their writing was measured by lexical analysis 
(lematati zation, text length, and word rareness). The essays of French 
students of English and of American students studying in France were 
compared. Results did not show text length to correlate with 
vocabulary richness. However, results indicated that a low, but 
significant, correlation existed between discrete-item test scores 
and lexical richness variables in each homogeneous group. (MSE) 
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THE LEXICAL RICHNESS OF L2 WRITTEN PRODUCTIONS AND THE VALIDITY 
OF VOCABULARY TESTS 



Pierre J.L. Arnaud (University Lyon 2) 



1 Introduction 

Discrete-item tests of separate components of language, although 
still in use for practical reasons, have tended to fall out of fashion 
as language testing theory has begun to place more emphasis on validi- 
ty. It seems* however, that vocabulary tests should be distinguished 
from the other categories. 

In terms of content validity, even keeping in mind the often 
repeated caveat that a vocabulary is not a "bag of words". It is clear 
that words are by and large Individual entities, and therefore that the 
lexicon lends Itself well to the construction of discrete Items. Fur- 
thermore, some types of vocabulary tests, such as those in which the 
testees are required to produce the name of Illustrated objects or ac- 
tions, correspond to the naming or labelling activity which is a fre- 
quent part of language behaviour In everyday life, and are Immune from 
the accusation of eliciting only "language- like behaviour". 

No reproach can be formulated against vocabulary tests on grounds 
of concurrent validity, either. In Ingram*s (1968) words, the vocabu- 
lary test is "the nearest thing we have to a foolproof test, i.e. a 
constructor-proof test. The statistics, both item analysis and cor- 
relation figures, are nearly always good." 

However, the ultimate proof of the validity of a test Is external: 
do test scores, obtained In an artificial situation, reflect accurately 
the quality of language behaviour in real-life circumstances ? The 
purpose of the present research Is to answer this question by comparing 
vocabulary test scores to Indices of vocabulary richness of L2 pro- 
ductions. 



2 Previous research 

Vocabulary richness has been Investigated intensively in lexi- 
cometric and stylos tat 1st leal research. This can provide useful metho- 
dological guidelines. In spite of the fact that the texts examined 
bear little resemblance to the productions of foreign-language 
learners. 

By vocabulary richness, one usually understands the number of 
types (V) in a given length of text. I.e. In a given number of occur- 
rences or tokens (N). Nuller (1977:115) specifically excludes any other 
measure, such as the degree of rareness of the vocabulary, which is 
not a "fact of structure", but a "fact of content". The comparison of 
texts of different lengths In terms of lexical richness poses a number 
of problems, as new words occur in decreasing numbers as a text un- 
folds Itself. This can be explained by means of the "urn model": 
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the lexicon of a writer can be compared to an urn, from which a 
lexeme/type Is drawn each time a token Is produced In discourse; the 
type Is then placed back Into the urn, from which It can be drawn 
again. Supposing the lexicon of a writer Is not Infinite, there remain 
fewer and fewer lexemes that have not been drawn before. An Index such 
as V/N Is therefore unstable. Other Indices for the comparison of 
texts of different lengths have been experimented with, but they have 
proved disappointing (Menard 1972:105f.; Muller 1977:117; Dugast 1978). 
In order to make legitimate comparisons. It Is therefore necessary to 
shorten the longer text, which can be done by actually deleting tokens 
or, alternatively, by using Huller's method of simulated draws based on 
the binomial law: knowing the distribution of frequencies for a given 
length of text. It Is possible to calculate the theoretical V 1 for an 
Inferior length (Muller 1977:101f .,127f .) . it should be noted, however, 
that the confidence Interval of V becomes wider as the original text 
Is more radically shortened. 

Menard (1972,1978) has Investigated the relationship between lex* 
leal richness, as defined above, and the presence of rare words In a 
text. After comparing the effects of various empirical norms of rare- 
ness, he concludes that too restrictive a definition (such as absence 
from a dictionary) leads to a Polsson distribution of rare words, thus 
making statistical treatment difficult. When more liberal norms are 
applied, highly significant rank correlations appear between lexical 
richness and the proportion of rare words In V (1). 

Less attention has been devoted to lexical richness by applied 
linguists and the author has found only two publications on the sub- 
ject: Llnnarud (1975) and Mendelsohn (1981). Llnnarud, In a pilot 
study, examines a corpus of 36 essays written In English by Swedish 
students; In addition, three native speakers, enrolled In the same 
course, wrote an essay on the same subject. As the author herself 
points out, the choice of the subject (" Sir, I protest... ") was un- 
fortunate as It led to very divergent responses and tnus to poor 
control of vocabulary content. The essays varied In length from 124 
to 573 words. Two main variables were taken Into account: lexical 
density and lexical variation. Lexical density Is fche percentage of 
lexical words In the text: LD » (N. * 100) / N. Llnnarud mentions 
difficulties due to the lack of Ie * firm criteria for distinguish- 
ing lexical words from grammatical ones, and the need to make care- 
fully weighed decisions In some cases. As some essays had a high LO, 
but contained a poor, repetitious vocabulary, lexical variation was 
also measured, with the following formula: LV » (V, * 100) / N* . 
Measures were co!1ec:ed once with errors deleted and once 
with errors retained; only the first series is reported here. The 
lexical density statistics (minimum: 30.86; mean: 38.51; maximum: 
47.15) fall below the values observed by Ure (1971) on LI texts* The 
LO of the essays written by native speakers Is higher. As the diffe- 
rent lengths of the essays made their comparison In terms of lexical 
variation Illegitimate, Llnnarud divided them Into four groups accord- 
ing to length, and within each group classified the essays Into three 
lexical variation categories: here, too, the native speakers scored 
better; their essays also contained a higher percentage of ha pax . 
Finally, the essays were classified Into three groups according to 
their global value, and groups were compared In terms of mean length: 
the "poor" essays are the shortest, but the "average", and not the 
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"good" ones, are the longest. However, the simple statistical treat- 
ments used In this pilot research provide suggestions more than defi- 
nitive results. 

Mendelsohn (1981) studied the oral productions of 42 subjects, 
native speakers of English and learners. He measured three variables 
that, according to him, constitute lexical richness. Lexical density 
was established by means of a formula slightly different from Llnna- 
rud's, but which would yield the same rank orders: LO - (N lex « 100)/ 
Nqr. Like Llnnarud, Mendelsohn mentions the difficulties due to the 
lexical /grammatical distinction, and recommends keeping notes of deci- 
sions ssade In ambiguous cases. Lexical variation was also measured 
using a slightly different method: LV - (Vi ex * 100) 7 N. A "semantic 
variation 11 measure was Introduced by Mendelsohn: when students are 
required to speak on a specific subject r a certain number of "universes 
of discourse" can be expected to appear In their productions, and the 
number of types belonging to them can be counted. Mendelsohn found 
that this was the most discriminating of the three measures of lexical 
richness. In addition, he measured the percentage of errors. As LO and 
LV yielded a rank-correlation coefficient of .78 and error percentage 
and lexical richness one of only .395, the author concludes that lex- 
ical richness and errors represent different dimensions of lexical 
performance, and that both should be taken Into account for evaluation 
purposes. In most situations, however. It seems that errors are the 
only criterion applied: observation of a group of 35 evaluators work- 
ing on the corpus showed that subjective judgments were mainly based 
on them, and that lexical richness had little Influence. Indeed, an- 
other group, required to classify subjectively a set of transcripts 
for lexical richness, performed poorly on the task. 

Other studies dealing with the assessment of written productions 
of L2 learners have yielded Interesting results concerning the role of 
vocabulary In the formation of judgments. Perkins (1981), comparing 
three methods of essay evaluation: global subjective, analytic subjec- 
tive (I.e. a group of judgments on criteria such as relevance, fluency, 
grammar, mechanics, vocabulary), and objective (number of errors, num- 
ber of words per T-unlt) found that, among the five subjective crite- 
ria, vocabulary had the lowest correlations with the other variables 
and the subjective global score. On the contrary, Mullen (1980), work- 
ing on a corpus of 117 texts assessed by four judges globally and 
analytically along four dimensions, structure, organization of Ideas, 
quantity of text produced and vocabulary, found that vocabulary was 
best correlated with the global score. 

Finally, the relationship between essay length and quality has 
been Investigated. Linnarud's (1975) results have already been quoted. 
Larsen-Freeman and Strom (1977) and Neumann (1977), as quoted by Per- 
kins (1980), found a significant correlation between essay length and 
overall quality, whereas Perkins himself did not observe such a rela- 
tionship. Again, Mullen (1980) reports a high correlation between 
quantity of text and global score, but this quantity Itself was esti- 
mated subjectively. 
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3 Method 

3.1 Administration 

The subjects were first-year specialist students of English at 
Lyon 2 University, in order to reach a sufficient population, the ex- 
periment was carried out during normal teaching hours. The vocabulary 

ad "i n1st « r ? d f1rst : U a 26 - 1te * Productive test (trans- 
lation from French Into English) with a reliability (K.R./Horst) of 

l9B \ h 00 the f °] l0 * 1n 9 week, an hour was devoted to the 
essay writing. To ensure motivation, global scores were taken Into 
account for continuous assessment. The subject was chosen so as to 
give every student something to say: 

^Ll! 0 ** 0 " S ,?? C ?V° 1"P«ve secondary education In France ?" 
2f inoTiJ?? 1 P° ssi ^ 111 t1es led to the retention of a random sample 
te /ess ?V air J' I" addition, four sophomores from Dartmouth 
X at Lyon 2 on a regular programme, wrote an essay on the 
following, symmetrical subject: 

r. M 5*! d ° you suggest to Improve high school education In the 
U.S.A. ? 

Although this control group Is too small to allow for statistical com- 
parisons, and In spite of the fact that the local students were hard- 
ly comparable to those In academic terms, a useful reference is thus 
provided. 

3.2 General considerations 

*e *h The e " ay * **rV 1rst rev1ewed subjectively. An essay can be seen 
E*n5^!^ un 5J l0 Vr< a numb€r of 1deas and thc1r connections on one 
.tiltl i i^Ji 109 " 1 ^ ■ tans t0 ««Pr«ss them on the other hand. There 
exists a much larger degree of Independence between the two In the case 

.IfI?irV ha \ 1r \ tne case of nat1ve speakers, and on examining a 
poor essay It Is not always easy to determine what Is due to a paucity 
rLl l S ? r t0 !J W lan 9^age proficiency. Furthermore, when one tries to 
52 ™! he P urely . 11n 9"1st1c aspects, clear-cut components sel- 

an^ess^ Jn P S?rS 6 ^ ly: f0 J 1nstance » one ™™ "a^dly ever come across 
an essay in which the sentence grammar was Impeccable and the text 
grammar disastrous. The following sample Is typical of this kind of 
pro Diem: 

"We should find other way for holiday. The summer holidays are too 
ong. During often three months, the pupils forget" what the y have 
learnt In the year. It should be better to have more holiday but 
shorter holiday and to finish the school at 4 o'clock and no t at 
Q clock . — — — . 

Such f peOITon might be attributed to lack of vocabulary or Ignorance 
?n ShM 1 ?? d ? VlC ? S ' ^ t t ie fra * nrnt ******* t0 * strikingly similar 
D?4SeJ°(lg8l:54S? r ****** discussed by de Beaugrande and 

"There's water through many homes. I would say all of them have 
TU " a * er ^mT It's just completely under water." 
The auffiors suggest that this discourse Is due foTack of planning 
time and rapid loss of the surface text, two phenomena which obviously 
play a role when students at this proficiency level have to write 
i52l SL s 1n a 11m1ted t1roe - essay Including many repetitions 
like the one quoted from above would result In • very low lexical 
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variation score, which would however to « **t extent be the effect 
of non-lexical causes. This should serve as 4 binder that total 
Isolation of one component Is probably lllusor* and that om should 
not place excessive trust In objective measures of written va- 
riables. 

A number of facts about lexical quality do appear clear:/, jm** 
ever. As Impressions are produced essential lyT>y lexical words* non- 
lexical tokens can be left aside, as In Llnnarud's (197S) and Mendel- 
sohn's (1981) studies. Some of the essays strike the reader as con- 
taining a very limited number of lexical Items repeated Interminably; 
this Is so widespread a phenomenon, and one that lowers the overall 
quality of an essay so much that It must be taken Into account In any 
definition of 12 lexical richness. As Mendelsohn (1981) had found that 
lexical density and lexical variation are highly correlated. It was 
decided to determine lexical variation through the following formula: 

Vlex 

LV « 

N 

In other cases, attention Is attracted to the vague, general vocabulary 
used by the writer. For Instance, the only way some students were able 
to express value judgments was literally to use good and not good . 
What Is Involved here Is the degree of rareness of the vocabulary and 
It was therefore decided to assess It. 

Finally, the number of lexical errors Is sometimes so high as to 
hinder communication. Naturally, the author Is well ar re that, as a 
non-native speaker of English, he may be more Intolerant of errors than 
a native speaker: James (1977) found that It Is the non-natives who are 
less tolerant of lexical errors; Hughes and Lascrratou (1982) have 
shown that non-natives react to the fundamental nature of the rules 
that are transgressed, whereas native speakers are sensitive to the way 
communication is affected. Mendelsohn (1981), as seen above, has 
brought to light the Importance of errors In the subjective evaluation 
of lexical quality. The present research therefore differs from clas- 
sical stylistic studies of LI texts In that It Includes the measurement 
of errors and vocabulary rareness. 

3.3 Data treatment 

3.3.1 Editing norms 

The texts had to bt treated on a small computer (2) with programs 
written by the author, and this required great simplicity In the ope- 
rations, which made the following editing norms necessary. 

Proper nouns f except country names) and numbers were deleted. 

Compound words : after their existence as Items was checked In a 
dictionary (Hornby 1974), they were typed hyphenated and thus treated as 
units by the program. 

Verb-particle constructions were typed separated or hyphenated 
according to tne case: 
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He turned down the street. 
He turned-down the offer. 

they -rl^TvlsrySrTS no l of direct relevance to this research, 

but N is som l.eT.&cfor* ™" S <n thC SCh0 ° U ' 

T . ^ The philosop hy Is not properly taught . 

If wluTJSlrSl?! <"POss.bilUy to Isolate lexical asp*<t* 

Jffif£l°? 1cil err0rS corrected when It was clear that 
ledge or the lexeme waslibt affected (but see below). 

iwdc- 6 ^^ 1 err ? rs i * 1tho<i 9 n »■ not concerned here with m < 
lysis, sucn regularities occurred that a brief survey Is of t# 
note the prevailing Influence of Interference. 

•Inor spelling Mistakes: perso nnal . teatcher 

•*1or spelling Mistakes: scholT = 

nation Mistakes: to cbMparate, he suc cessed 

is d1fTlcult KtPt1Ve C09nit * s * : **** sh oulo be pre* 

Interference fro* another language on tm cvrlculu*. *v > 
Money - 

AlthnSSi 0 " ^J!**"/? 0 . ,e "*" : Ij^»cnersjearn the. tt 
««^ 9 th f 9rav1ty °f J t 5 BS- ^P" oTtrBSTT w variab ly "in . - 
wUh .2 ^ST SI ,pP I 1ed: ,ny '"^""y erroneous for. was re, ... 
with an error code, and was counted In N but not In V. 

fn^ffJ^S!!'""? cases l S0Be "orphologlcal errors Indicate that the 

SSTte^ES 'f T ? t0 S d .? Orr-ct, y ! erro » on Irregular 

verts were treated as lexical. Sl.llarly, the preposition In the 

SIS" n h?t e I 4 ? Pl Lr a ? tre,ted < " a lex1 "' -1«tffi a we cannot con- 
wstereo kn ° W " ' th * """Wl". structures are not 

They listen a t the teacher , 
otherwise treated as grammatical Items. 

thp w"?l!" s g u ?^° k the J Sub J ec 4 ! the lack of correspondence between 
SwhlE C S!:-" rU J s !I a ? a Fr 5 Bch educ4t1 onal syste«sTaused a number of 
Sn?I^t; * K tUdC ?L use ^ a ter " fro " an Wo-saxon context to 
designate a French reality, this was not considered as a .1 stake, but 
the word was Ignored. Furthermore, It was often difficult to determine 

errors t^!^ "V" Wa M e 9"<~i« or constituted an EterfcSnce 
ZZXZl as . the con text "« ambiguous: for Instance, It was seldou clear 
whether mjnlster stood for minister; or .Inlstre . and If class was 1^ 
iTttl 5° !* an Hi" or c assroom (both cU«se 1c , French)7The benefit 
Urt^# d0Ubt «*1* s y st «*^caTTyTccorded7T ~th1s Introduced a farther 
lack of precision In the measurement. 
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3.3.2 Lexical vs. grammatical xords 

For automatic treatment r wrposes, It was out of the question to 
have to Make a decision every time an ambiguous fom turned up; to* 
kens like have or do were therefore systematically treated as gramma- 
tical. As FTTlterTor non-lexical occurrences, the program Included 
a dictionary of grammatical forms to which each token was compared In 
turn. As treatment time grew exponentially with the size of this dic- 
tionary. It had to be limited to approximately 100 Items. These are 
the first grammatical Items 1* the rank-order list of the Hofland and 
Johansson (1982) frequency count. The classical closed class/open 
class distinction was therefore not respected, as many closed-class 
members Mere left outside the dictionary, but this was not really a 
weakness, for few of the grammatical Items present In the corpus passed 
throiyh the filter, given the proficiency level of the subjects; 
furthermore, knowledge of a rare grammatical word cannot be considered 
as Independent from lexical knowledge. 



3.3.3 Lemmatizatlon 



Lemmatlzatlon was carried out manually on the print-outs. Plurals 
were grouped together with the singulars, conjugated forms with the 
Infinitive, comparatives and superlatives with the positive degree. 
Only one lemma was counted for forms belonging to two categories 
( book's , booked). 

3.3.4 Lengths and Indices 

The distribution of essay lengths Is represented In Figure 1. Va- 
riation Is considerable. 



Figure 1 

distribution ot N 
(essay lengths) 

H - 313 

SV « 91 

(N Ss ; 100 ♦ 4) 
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figure 2 

CVoluilon of V/N and other Indices: successive 50-word 
fragments of two essays 
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t ioo 2CO 300 400 500 

log V N ♦ 

log n 



The graphs of Figure 2 prove empirically that V] e «/N and Indices de- 
rived from It are unsuitable for the estebl i shment of lexical varia- 
tion scores on L2 texts In this length range. It lies therefore neces- 
sary to bring dot* all the essays to the length of the shortest one. 
N • 180. Use of Killer's (1977) method of simulated draws would have 
been exceedingly time-consuming, so token deletion was chosen In- 
stead* It was impossible to retain* for Instance, the first 160 to- 
kens of each essay as this would not have ensured adequate sampling. 
A series of Instructions was therefore added to the orogram for the 
random selection of 180 tokens. The lexical variation score hence- 
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forth corresponds staply to V. and tht trror score (C) to tht nu»- 
btr of lexical trrors 1a tht sanplt. 

3.3.S Mora of rareness 

It would havt been theoretically possible to calculate tht ave- 
rage frequency of tht types 1n each essay, using a frequency list, 
but such a task Mould have proved unsunaountable practically* An 
enplrlcel solution Hit Mtnard's (ItTi) was t hart fort retained, and 
tht nora was chest* as trldt as posslblt. Tht official French Ministry 
of Education vocabulary 11st for tht classt dt trolsUg (elaboration 
1976) was ustd. It contains I S22 lexical VBm IBI art suppostd to 
bt known of all studtnts by tht tint thty rtach tht Igtt level. A 
type was thtrtfort consldtrtd as rart 1f 1t was not onTne 11st. Tht 
following fonwila was ustd for establishing tht rartntss scorts: 




3.3.6 Mtl1abU1t1ts 

As text shorttnlnf Introducts a asrgln of uncertainty, tht reHa- 
bU1t1ts of V, R end E wtrt checked. Tht progrea was run four tints over 
a seaplt of 20 assays, tach tint extracting a dlfftrtnt stt of 160 to* 
tats. Product non tn t cotff1c1tnts of correlation wtrt coaputcd bttwttn 
tht valuta yltldtd by rurs I and II. and III and IV respectively. Tht 
rt suits of tht first and last two runs wtrt than averaged, and tht cor* 
rotations bttwttn tht two stts of atens calculated. Tht Hgurts thtrt- 
fort corrtspond to reliability cotffldtnts for valuta obtained afttr 
two runs of tht prooraa, which was tht cast for tht remaining 80 (*4) 
assays (Table 1). 



Tablt 1: reliability cotffldtnts of lexi- 
cal Hchntss variables afttr ttxt shorten- 
ing; bottoa Hot: r's for nean valuta af- 
ttr two runs of tht progrea (20 texts) 



V A i 

I II III IV I II III IV I II III IV 

M .73 .86 .90 .76 .97 

.87 .96 .92 
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3.3.7 Distributions 

The distributions of V, R and E are reproduced In Figures 3, 4 and 
5. Interestingly, In terms of lexical variation and rareness of vocabu- 
lary, some French students performed as well as their American counter* 
parts. The error scores of two of the latter are due to spelling mis- 
takes. 
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Figure 5: distribution of £ 
4 Results and discussion 

It is not unreasonable to think that the more proficient students 
are able to write longer texts in a limited time. Subjective examina- 
tion, however, had shown that some of the longest essays were among 
the most repetitive and unconnected. This impression is confirmed, as 
far as vocabulary richness Is concerned, by the correlation coeffi- 
cients in Table 2, none of wtfch is significant. In view of the con- 
tradictory results of previous research (see section 2), It seems that 
no generalizations should be made in this respect. 

Table 2: correlations between essay length 
and the three lexical richness variables 
as established on 180-token samples 



r NV 
r NR 
r NE 



.06 
.10 
.05 
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Product -moment coefficients of correlation between test scores 
(T) and the three lexical richness variables are reproduced In Table 3. 
There obtain significant, but low correlations between V and R and V 
and E (3); R appears to be uncorrelated to T and E. However, It would 
be unwise to conclude that R r.ieasures a differ .nt dimension of lexical 
richness before replication studios can be carried out. 



Table 3: correlations of test scores (T) and lex- 
ical richness variables (N - 100); * : p<.025; 
: p<.0l 5 



T 
V 
R 



V 

.36" 



R 

.09 
.27* 



E 

-.21* 
-.24* 
.09 



At this stage, a few points should be remembered. First, a test 
score is the result of a concentrated measurement situation, which Is 
not the case of an essay. Secondly, a substantial amount of uncertainty 
was Introduced successively by editing norms, decisions as to errors 
and text shortening; absolute Isolation of the lexical component was 
furthermore Impossible. Finally, the students, all native speakers of 
French with fresh b accalaurfeats , constituted a very homogeneous group 
and this obviously" lowered statistics (see Raatz and Kleln-Braley 
1981). In view of these considerations, the low, but significant corre- 
lations between T and V and T and E make It possible to conclude that 
discrete-item vocabulary tests are valid. 



5 Suggestions for further research 

One of the questions njhich It has not been possible to answer in 
this study Is that of the concurrent validity of the lexical richness 
measures. The only way to do this would be to analyse two sets of es- 
says written by the same subjects, which was administratively beyond 
the author's reach. 

For further research, use of the following empirical index is 
suggested: 

LR " V lex + V r.re " 2E 

(on text samples of the same length). Rare types and errors are given 
double weight. This Index was found to separate native speakers from 
learners very clearly. 
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Notes 

1 Mfrnard, however, does not prove that an Index like V / V Is 
Independent from the size of V. rarc 

2 The author wishes to thank his colleagues of the Service In forma - 
tique. University Lyon 2 for providing unlimited access to the 
computer and indispensable advice. 

3 See Note 1 for a possible dependence between R and V. Further- 
more, as error words are at the same time counted In E and Ig- 
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